What is feature engineering?

1. conversion of raw data into form amenable to ML estimation \(\implies\) data munging
2. designing features that have predictive power \(\implies\) data visualisation

1 is necessary for all ML models; 2 is most necessary for non-deep learning models

Example dataset: California house prices (Kaggle)

Aim: develop model to predict house prices

Examine data (note this is at block level)

Missing features

Handling missing data: options

• drop rows with missing obs.
• drop columns (i.e. variables) with lots of missing obs.
• impute observations

Imputation

• lots of options
• simple method: use median (for continuous data) or mode (for categorical / ordinal)
• model dependencies: regress variables on all other variables and use this
• models incorporating uncertainty: useful if substantial portion missing

How to handle categorical data?

Median income by ocean proximity

One-hot encoding

House price versus continuous attributes

Income vs house price

Handling censored observations

• leave them as they are
• or could drop censored observations altogether
• (better) impute them
• (better still) explicitly model uncertainty in them

Feature creation: rooms per household

Bedrooms per room

Persons per house

Feature scaling

• ML algorithm training is much improved if features on similar scales
• rescaled features can also better represent relationships between variables

House price rescaling (ignoring upper limits)

Option: standardise

Subtract mean and divide through by sd

Option: normalise

Option: log transform

Feature scaling: no golden rule

• graph features before/after scaling: outliers can seriously affect scaling
• build feature scaling options into ML pipeline
• evaluate impact of different scaling options on test set prediction

Importance of pipelines

• many options for imputation, feature creation, feature scaling
• ML software, for example, Scikit-learn in Python has great options for creating data processing pipelines incorporating these
• pipelines make it easy to cleanly (and without error) process data
• and to test out different options

Questions?

Metrics for regression

most common, root mean squared error:

\[\begin{equation} \text{RMSE} = \sqrt{\frac{1}{K}\sum_{i=1}^{K} (\hat{y}_i - y_i)^2} \end{equation}\]

also used:

\[\begin{equation} R^2 = \frac{\text{variation explained by model}}{\text{total variation}} \end{equation}\]

Metrics for binary classification: confusion matrix

True positive and false positive rate

Classification boundaries

classifiers output class probabilities. For example,

\[\begin{equation} \text{Pr}(y_i=\text{cat}|x_i) = 0.4 \end{equation}\]

an obvious choice for classification boundary is \(\text{Pr}(y_i=\text{cat}|x_i) = 0.5\). But this doesn’t work well for imbalanced datasets. Want good performance across all boundaries

Classification boundaries

vary boundary cutoff value and calculate true positive rates (TPR) and false positive rates (FPR).

When \(\text{Pr}(y_i=\text{cat}|x_i) = 0.0\) is boundary \(\implies\) all positive:

• \(\text{TPR}=\frac{\text{TP}}{\text{TP}+\text{FN}} = \frac{\text{TP}}{\text{TP} + 0} = 1\)
• \(\text{FPR}=\frac{\text{FP}}{\text{FP}+\text{TN}} = \frac{\text{FP}}{\text{FP} + 0} = 1\)

When \(\text{Pr}(y_i=\text{cat}|x_i) = 1.0\) is boundary \(\implies\) all negative:

• \(\text{TPR}=\frac{\text{TP}}{\text{TP}+\text{FN}} = \frac{0}{0 + \text{FN}} = 0\)
• \(\text{FPR}=\frac{\text{FP}}{\text{FP}+\text{TN}} = \frac{0}{0 + \text{TN}} = 0\)

Random classifier: ignores rows and (here) assigns 1/4 to cats

ROC and AUC: random classifier

ROC and AUC: random classifier

Applying a gradient boosted machine

• use a gradient boosted machine (a form of decision tree) to predict house price
• use lots of deep trees
• (on a subset of full data)

Result

Performance of model on independent data

How to tune hyperparameters?

• more complex models \(\implies\) explains more variation in a given dataset
• some variation is idiosyncratic nuisance variation \(\implies\) fits noise not signal
• instead, fit data to training set and evaluate performance on separate cross-validation set
• use cross-validation set performance to determine optimal hyperparameters

Train and cross-validation set performance

Cross-validation approaches

• using a single train / CV set risks tuning to the noise in the CV set
• solution: use many train / CV sets along subsets of your data
• example: k-fold cross validation

3-fold cross-validation

How many folds?

How to assess predictive accuracy?

• could state predictive accuracy as that achieved on CV set \(\implies\) overestimate
• instead hold aside a separate test set on which a model is evaluated only once
• make test set similar to task model eventually used for
• typically something like 70%-90% train + CV and leftover is test

Workflow

Questions?

Data visualisation

• in statistics in general no important result should come as a (total) surprise
• especially true in ML
• should know your problem well enough before starting ML proper
• \(\implies\) data visualisation key
• (also building simpler more understandable models first can help)

Variable importance

• after fitting model \(\implies\) good to know which variable drives performance
• straightforward for simple models like linear and logistic regression
• harder for more black-boxy models but methods exist

Example: RF variable importance

Which model to choose when

• simpler models can be useful to guide complex models
• use performance on validation set to guide model choice
• for rectangular data \(\implies\) RFs and gradient boosted models like XGBoost best
• for data with richer structure \(\implies\) deep learning

Golden rules of supervised ML

1. understand your data: visualise
2. pre-process data and design useful features; build pipelines
3. determine what is a good baseline accuracy and, if possible, target accuracy
4. make sure you align CV and test sets
5. know how your ML model works; what its hyperparameters mean
6. choose hyperparameters via a grid search
7. examine literature for domain level model choice and hyperparameter choice

Questions?